Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

📊 Replace backported dataset by proper GHO #3171

Merged
merged 2 commits into from
Aug 22, 2024
Merged

Conversation

Marigold
Copy link
Collaborator

No description provided.

@owidbot
Copy link
Contributor

owidbot commented Aug 22, 2024

Quick links (staging server):

Site Admin Wizard

Login: ssh owid@staging-site-cholera-refactor

chart-diff: ✅ No charts for review.
data-diff:
= Dataset garden/who/2023-06-01/cholera
  = Table cholera
    ~ Column cholera_case_fatality_rate (changed data)
        ~ Changed values: 2 / 2992 (0.07%)
          country  year  cholera_case_fatality_rate -  cholera_case_fatality_rate +
            World  2016                      1.831696                      1.831765
             Asia  2016                      0.997723                      0.997993
    ~ Column cholera_reported_cases (changed data)
        ~ Changed values: 3 / 2992 (0.10%)
          country  year  cholera_reported_cases -  cholera_reported_cases +
             Iraq  2016                         5                      <NA>
            World  2016                    132118                    132113
             Asia  2016                     18442                     18437
2024-08-22 09:57:10 [error    ] Traceback (most recent call last):

  File "/home/owid/etl/.venv/lib/python3.10/site-packages/requests/models.py", line 974, in json
    return complexjson.loads(self.text, **kwargs)

  File "/home/owid/etl/.venv/lib/python3.10/site-packages/simplejson/__init__.py", line 514, in loads
    return _default_decoder.decode(s)

  File "/home/owid/etl/.venv/lib/python3.10/site-packages/simplejson/decoder.py", line 386, in decode
    obj, end = self.raw_decode(s)

  File "/home/owid/etl/.venv/lib/python3.10/site-packages/simplejson/decoder.py", line 416, in raw_decode
    return self.scan_once(s, idx=_w(s, idx).end())

simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0)


During handling of the above exception, another exception occurred:


Traceback (most recent call last):

  File "/home/owid/etl/etl/datadiff.py", line 423, in cli
    lines = future.result()

  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 458, in result
    return self.__get_result()

  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception

  File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)

  File "/home/owid/etl/etl/datadiff.py", line 416, in func
    differ.summary()

  File "/home/owid/etl/etl/datadiff.py", line 254, in summary
    self._diff_tables(self.ds_a, self.ds_b, table_name)

  File "/home/owid/etl/etl/datadiff.py", line 122, in _diff_tables
    table_a = future_a.result()

  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 458, in result
    return self.__get_result()

  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception

  File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)

  File "/home/owid/etl/.venv/lib/python3.10/site-packages/tenacity/__init__.py", line 336, in wrapped_f
    return copy(f, *args, **kw)

  File "/home/owid/etl/.venv/lib/python3.10/site-packages/tenacity/__init__.py", line 475, in __call__
    do = self.iter(retry_state=retry_state)

  File "/home/owid/etl/.venv/lib/python3.10/site-packages/tenacity/__init__.py", line 376, in iter
    result = action(retry_state)

  File "/home/owid/etl/.venv/lib/python3.10/site-packages/tenacity/__init__.py", line 398, in <lambda>
    self._add_action_func(lambda rs: rs.outcome.result())

  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result
    return self.__get_result()

  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception

  File "/home/owid/etl/.venv/lib/python3.10/site-packages/tenacity/__init__.py", line 478, in __call__
    result = fn(*args, **kwargs)

  File "/home/owid/etl/etl/datadiff.py", line 837, in get_table_with_retry
    return ds[table_name]

  File "/home/owid/etl/etl/datadiff.py", line 278, in __getitem__
    return tables.load()

  File "/home/owid/etl/lib/catalog/owid/catalog/catalogs.py", line 312, in load
    return self.iloc[0].load()  # type: ignore

  File "/home/owid/etl/lib/catalog/owid/catalog/catalogs.py", line 363, in load
    return Table.read(uri)

  File "/home/owid/etl/lib/catalog/owid/catalog/tables.py", line 178, in read
    table = cls.read_feather(path, **kwargs)

  File "/home/owid/etl/lib/catalog/owid/catalog/tables.py", line 350, in read_feather
    cls._add_metadata(df, path, **kwargs)

  File "/home/owid/etl/lib/catalog/owid/catalog/tables.py", line 322, in _add_metadata
    metadata = cls._read_metadata(path)

  File "/home/owid/etl/lib/catalog/owid/catalog/tables.py", line 384, in _read_metadata
    return cast(Dict[str, Any], requests.get(metadata_path).json())

  File "/home/owid/etl/.venv/lib/python3.10/site-packages/requests/models.py", line 978, in json
    raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)

requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

= Dataset garden/who/2024-02-14/gho_suicides
  = Table gho_suicides_ratio
  = Table gho_suicides
= Dataset garden/who/2024-05-20/vehicles
  = Table vehicles

⚠ Found errors, create an issue please

Legend: +New  ~Modified  -Removed  =Identical  Details
Hint: Run this locally with etl diff REMOTE data/ --include yourdataset --verbose --snippet

Automatically updated datasets matching weekly_wildfires|excess_mortality|covid|fluid|flunet|country_profile|garden/ihme_gbd/2019/gbd_risk are not included

Edited: 2024-08-22 09:57:13 UTC
Execution time: 487.06 seconds

@Marigold Marigold marked this pull request as ready for review August 22, 2024 07:05
@Marigold Marigold merged commit d2ff4f5 into master Aug 22, 2024
8 checks passed
@Marigold Marigold deleted the cholera-refactor branch August 22, 2024 10:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants